fix: reuse CN pipeline connections after clean teardown#25713
Conversation
Negotiate FIN/FIN_ACK before retiring morpc stream state, coordinate server cleanup with client reuse, and add rollback metrics plus unhappy-path acceptance coverage.
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Coverage gap fixed in |
Merge Queue Status
This pull request spent 17 seconds in the queue, including 3 seconds running CI. Required conditions to merge
|
What type of PR is this?
Which issue(s) this PR fixes:
Closes #25690
Related to #25691.
What this PR does / why we need it:
Remote pipeline completion currently couples a logical stream teardown to destruction of its morpc backend and CN-to-CN TCP connection. High-fanout distributed queries therefore create enough connection churn and
TIME_WAITtuples to exhaust Linux ephemeral ports even though the CN processes remain healthy.This PR adds a negotiated FIN/FIN_ACK lifecycle for successfully drained pipeline streams:
disable-stream-reuseprovides a rollback switch.The connection remains exclusively locked while a logical stream is active. This PR does not multiplex active streams on one TCP connection, add query-boundary admission, or change the existing maximum-concurrency policy. Those require separate deadlock/fairness design under #25691.
Validation
go test -count=1 ./pkg/common/morpc ./pkg/cnservice ./pkg/sql/compile-racemake -j8 build499999; CN pipelineTIME_WAITremained zeroTIME_WAITremained zero during load; idle GC returned established pipeline sockets to zero and CN file descriptors from about 2,895 to 63499999after idle GCThe design, failure matrix, rollout/rollback plan, and detailed acceptance records are included in
docs/design/cn_pipeline_connection_lifecycle.md.